ENH: OOC architecture rewrite — new bulk I/O API and infrastructure#1568
Open
joeykleingers wants to merge 1 commit into
Open
ENH: OOC architecture rewrite — new bulk I/O API and infrastructure#1568joeykleingers wants to merge 1 commit into
joeykleingers wants to merge 1 commit into
Conversation
b4ef97f to
99b49ed
Compare
b4ef97f to
bb09048
Compare
This was referenced Mar 24, 2026
102c436 to
b4c1358
Compare
2bd614a to
110c054
Compare
35aecd0 to
3a88bbf
Compare
bdfed87 to
6fbfc8d
Compare
2095f32 to
65c4ba8
Compare
65c4ba8 to
653cf62
Compare
1773602 to
4834fa1
Compare
Introduce a compile-time out-of-core storage capability for simplnx: a
DataStore abstraction that can back arrays on disk, an IO/recovery path
that registers and finalizes those stores, a memory-budget manager that
decides in-core vs OOC placement, and chunk-aware parallel dispatch so
algorithms stream chunked data efficiently. OOC is gated behind the
compile-time SIMPLNX_USE_OOC switch and is wholly absent from in-core
builds. Every OOC entry point is a direct SimplnxOoc:: function call
guarded by #ifdef SIMPLNX_USE_OOC and compiled into libsimplnx -- there
is no runtime hook, function pointer, or plugin indirection.
93 files changed, +5501 / -1229 lines.
================================================================================
1. Compile-time OOC switch and SimplnxOoc integration
================================================================================
Files: CMakeLists.txt, cmake/SimplnxConfig.hpp.in
Add the SIMPLNX_USE_OOC option and a generated SimplnxConfig.hpp that
carries the macro through simplnx's PUBLIC include path, so the value is
identical (ODR-safe) across every consumer and MOC. When enabled, the
private SimplnxOoc sources are compiled in place into libsimplnx -- no
copy into the build tree, no separate library, no simplnx->SimplnxOoc
link cycle -- and a source-dir guard fails configuration if the path is
missing. simplnx core calls SimplnxOoc:: functions directly, all behind
#ifdef SIMPLNX_USE_OOC.
================================================================================
2. DataStore layer: in-core / out-of-core abstraction
================================================================================
Files: src/simplnx/DataStructure/{AbstractDataStore,DataStore,EmptyDataStore}.hpp,
src/simplnx/DataStructure/EmptyStringStore.hpp,
src/simplnx/Utilities/{DataStoreUtilities,ArrayCreationUtilities}.hpp
Generalize the data-store interface so an array can be backed in core or
out of core. Array creation resolves its storage format with a direct,
compile-time-gated SimplnxOoc::resolveFormat call and builds a chunked
store via SimplnxOoc::createChunkedStore / createChunkedListStore when the
OOC format is requested; in-core builds compile none of these calls and
stay fully in core. Add an empty string store and extend the empty data
store for the deferred-load case.
================================================================================
3. IO collection, format registration, and recovery write
================================================================================
Files: src/simplnx/DataStructure/IO/Generic/DataIOCollection.{hpp,cpp},
src/simplnx/DataStructure/IO/HDF5/{DatasetIO,NeighborListIO,DataStructureWriter}.*
The DataIOCollection constructor calls SimplnxOoc::registerIOManager to
add the OOC IO manager, and finalizes stores on write via
SimplnxOoc::finalizeStores -- replacing the former plugin's load-time
setup with direct calls. The HDF5 writer calls
SimplnxOoc::maybeWriteRecoveryArray directly so dirty OOC arrays are
flushed during WriteFile.
================================================================================
4. DREAM3D file loading API and deferred load
================================================================================
Files: src/simplnx/Utilities/Parsing/DREAM3D/Dream3dIO.{hpp,cpp},
test/Dream3dLoadingApiTest.cpp, test/IOFormat.cpp
Rework the .dream3d loader API. In OOC builds, import is deferred via a
direct SimplnxOoc::handleImport call (lazy load) and writes run under a
SimplnxOoc::RecoveryWriteGuard; in-core builds load eagerly. Migrate the
compression test cases to the new loader API and add coverage for the
loading paths.
================================================================================
5. Memory budget management and the nxrunner --memory-budget flag
================================================================================
Files: src/simplnx/Utilities/MemoryBudgetManager.{hpp,cpp},
src/simplnx/Core/Preferences.{hpp,cpp}, src/nxrunner/src,
test/MemoryBudgetManagerTest.cpp
Add a MemoryBudgetManager that tracks the budget governing in-core vs OOC
placement. Preferences seeds the OOC base directory (via a direct
SimplnxOoc::setBaseDirectory call) and default format on startup and gains
a removeValue helper. nxrunner exposes a --memory-budget flag wired through
to the budget manager, with parsing hardened against NaN, inf, and
trailing garbage.
================================================================================
6. Chunk-aware parallel dispatch and the Extent utility
================================================================================
Files: src/simplnx/Utilities/AlgorithmDispatch.hpp, src/simplnx/Common/Extent.hpp,
test/{IParallelAlgorithmTest,ExtentTest}.cpp
Extend algorithm dispatch with direct vs scanline paths so chunked OOC
arrays are traversed in chunk-sequential order rather than thrashing the
cache. Add an Extent helper for region math. Both gain unit tests.
================================================================================
7. Filter, action, and test updates for OOC
================================================================================
Files: src/Plugins/SimplnxCore/.../Filters (+Algorithms, incl. FillBadData),
src/simplnx/Filter/Actions, plugin tests, test/UnitTestCommon
Propagate the chosen store format through the data-creation actions and
update the affected SimplnxCore filters/algorithms (notably FillBadData,
reworked to stream Z-slabs via bulk I/O) and OrientationAnalysis/
ITKImageProcessing call sites to the new APIs. Extend UnitTestCommon and
plugin tests accordingly.
Verified: the full OOC-Release build (libsimplnx with the SimplnxOoc
sources compiled in place, all plugins, and all test executables) compiles
and links cleanly; InCore-Release (SIMPLNX_USE_OOC=OFF) builds and the
DREAM3D-NX app launches. The ctest suite was not run.
Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
4834fa1 to
d4f634b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rewrites the out-of-core (OOC) architecture in simplnx, replacing the old chunk-based API with a new bulk I/O design built around
copyIntoBuffer/copyFromBufferonAbstractDataStore. Introduces the core infrastructure that the OOC-optimized filter algorithms (separate PR #1575) build upon.Core Architecture Changes
AbstractDataStore/IDataStore(loadChunk,getNumberOfChunks,getChunkLowerBounds,getChunkUpperBounds,getChunkShape)copyIntoBuffer/copyFromBufferpure virtual bulk I/O methods toAbstractDataStorewith implementations inDataStore,EmptyDataStore, andHDF5ChunkedStore(in SimplnxOoc plugin)StoreTypeenum (InMemory,OutOfCore,Empty) toIDataStore;IsOutOfCore()now checksStoreTypeinstead ofgetChunkShape()HDF5ChunkedStoreperforms I/O via HDF5 hyperslab selections with Z-slice-aligned default chunk shape{1,Y,X}for 3D datacopyFromBufferfast path: skips read-modify-write for tuple-aligned writescopyIntoBufferfast path: direct span-basedreadTuplesfor tuple-aligned readsDatasetIOgainsreadTuples/writeTuplesfor direct hyperslab-based bulk tuple I/ONew Core Utilities
DispatchAlgorithm— Runtime dispatch between in-core (Direct) and OOC (Scanline/CCL) algorithm variants based on data store typeSliceBufferedTransfer— Type-dispatched Z-slice buffered tuple copy utility that eliminates per-element OOC overhead during morphological transfer phasesUnionFind— Vector-based disjoint set data structure with union-by-rank and path-halving compression for chunk-sequential CCL algorithmsSegmentFeaturesOOC path — Z-slice CCL-based connected component labeling withUnionFindequivalence tracking, replacing BFS/DFS flood fill for OOC dataAlignSectionsOOC path — Bulk slice read/write withAlignSectionsTransferDataOocImplDataArrayUtilitiesbulk I/O —ImportFromBinaryFile,AppendData,CopyData, and mirrorswap_rangesupdated with chunked bulk I/O (runtime OOC check preserves original in-core performance)OOC Store Management
DataIOCollection/IDataIOManager— Updated for OOC store lifecycle managementImportH5ObjectPathsAction— OOC-aware file import with recovery metadataDataStoreIO— Detect OOC recovery attributes inReadDataStorefor safe data restorationTest Infrastructure
CompareDataArraysrewritten to usecopyIntoBufferin 40K-element chunks instead of per-elementoperator[]ForceOocAlgorithmGuardfor dual-path test coverageSIMPLNX_TEST_ALGORITHM_PATHCMake option (0=Both, 1=OOC-only, 2=InCore-only) for build-specific test path controlRelated PRs
Test Plan